Fix downgrade CI: split the AD test group out of the downgrade lane#59
Merged
ChrisRackauckas merged 3 commits intoJun 21, 2026
Merged
Conversation
The Downgrade (Core) job was failing with an Unsatisfiable resolution on Julia lts (1.10). When julia-downgrade-compat pins every dep to its lowest allowed version, FunctionWrappers was pinned to 1.0.0 (floor of "1"), but Mooncake (a tested weakdep/extension target) requires FunctionWrappers >= 1.1.3. That made the merged test environment unsatisfiable. Raising the FunctionWrappers compat floor from "1" to "1.1.3" matches the version actually exercised by the test extras and makes the minimum-version resolution satisfiable. Verified locally on Julia 1.10: the merged test project now resolves at minimum versions, the package loads, and basictests pass with FunctionWrappers 1.1.3. Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ools/TruncatedStacktraces floors
The Downgrade (Core) job was red on lts (Julia 1.10). Three distinct, genuine
problems, fixed at their causes:
1. Unsatisfiable resolution. The resolver (julia-downgrade-compat -> Resolver.jl)
minimizes Julia to the floor of its --julia range; on lts that range (1.10)
admits 1.10.0. But every Mooncake 0.5.x test extra requires julia >= 1.10.8,
so Mooncake can never be placed -> Unsatisfiable. Even at 1.10.8 the
Enzyme 0.13 + EnzymeCore 0.8 + Mooncake 0.5 minimum-version set has no
solution on the 1.10 LTS stdlib line; it resolves only once Julia reaches
1.11.6 (Mooncake 0.5's 1.11 floor). Resolver.jl ignores the project's
[compat] julia for the Julia version itself, so no Project.toml change can
force julia >= 1.10.8 -- the caller must request it. Fix: pin the Downgrade
caller to julia-version "1.11" (resolver picks 1.11.6). Core 1.10 support is
still exercised by the regular Tests workflow.
2. PrecompileTools floor too low. With resolution fixed, the minimum
PrecompileTools 1.0.0 fails to precompile GPUCompiler (pulled transitively by
Enzyme) on Julia 1.11: "UndefVarError: PrecompileTools not defined in
GPUCompiler". GPUCompiler's compat says PrecompileTools "1", but 1.0.0 does
not actually provide the API GPUCompiler uses; 1.1.0 does. Raise the floor
to "1.1".
3. TruncatedStacktraces floor too low. src uses
TruncatedStacktraces.@truncate_stacktrace, which does not exist before v1.1.0
("UndefVarError: @truncate_stacktrace not defined in TruncatedStacktraces" at
the v1.0.0 floor). Raise the floor to "1.1".
Also reverts the earlier misdiagnosed FunctionWrappers floor raise (1 -> 1.1.3):
no registered test dep constrains FunctionWrappers and the source only uses the
FunctionWrapper{R,A} constructor available since 1.0.0. Version bumped to 1.9.2
for the compat-floor changes.
Local verification on Julia 1.11 (the new downgrade leg): the actual
julia-downgrade-compat downgrade.jl resolves the merged test project (exit 0,
"Successfully resolved minimal versions"); the downgraded manifest (Enzyme
0.13.0 / GPUCompiler 0.27.5 / PrecompileTools 1.1.0 / TruncatedStacktraces 1.1.0
/ Mooncake 0.5.0) precompiles cleanly; and Pkg.test GROUP=Core passes
(FunctionWrappersWrappers.jl 48/48, BigFloat + UnionAll 7/7).
Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
de1b712 to
9544c12
Compare
The Downgrade (Core) red was caused by the AD test extras (Enzyme, EnzymeCore, Mooncake) being part of the root test target. The downgrade resolver (julia-downgrade-compat -> Resolver.jl) minimizes Julia to the floor of its --julia range, but every Mooncake 0.5.x requires julia >= 1.10.8 and the Enzyme 0.13 + EnzymeCore 0.8 + Mooncake 0.5 floor set has no minimum-version solution on the 1.10 LTS line, so the merged test env was Unsatisfiable (the Resolver.jl <-> Mooncake wall, StefanKarpinski/Resolver.jl#24). The AD deps shouldn't gate the downgrade lane. Fix: move the AD test deps into their own per-group sub-environments (test/Enzyme/Project.toml, test/Mooncake/Project.toml) and drop them from the root [extras]/[targets].test. SciMLTesting's run_tests already supports per-group sub-envs (the `env = ...` group form, as the nopre group uses); the Enzyme/Mooncake groups now declare their own env. The downgrade job runs GROUP=Core, which resolves only the root test target -- now AD-free -- so it is satisfiable on lts (Julia 1.10) again. The AD groups are added to test/test_groups.toml so they run as their own jobs in normal CI (lts, 1); previously they were declared in runtests.jl but had no matrix entry and so never ran. Reverts the SciML#59 accommodations that masked the AD-resolution problem: - .github/workflows/Downgrade.yml: julia-version back to "lts" (no longer needs to be pinned to 1.11 to dodge the AD floor set). - PrecompileTools compat floor back to "1": the 1.0.0 GPUCompiler precompile failure only arose because Enzyme pulled GPUCompiler into the downgrade env; with AD out of that env it no longer applies. Kept, because they are genuine minimum-version bugs independent of AD (both verified locally on Julia 1.10): - TruncatedStacktraces = "1.1": src uses @truncate_stacktrace, which does not exist before v1.1.0 (1.0.0 is a 25-line stub). - FunctionWrappers = "1.1.2": 1.0.0 / 1.1.0 / 1.1.1 fail to build the FunctionWrapper cfunction ("Module IR does not contain specified entry function") on Julia 1.10; 1.1.2 is the lowest that builds and runs. This is exercised by the package's own PrecompileTools @compile_workload, so it is a real floor regardless of the AD split. Local verification (Julia 1.10/lts, the actual downgrade leg): - julia-downgrade-compat downgrade.jl: exit 0, "Successfully resolved minimal versions for . (with extras)"; downgraded manifest contains no Enzyme/Mooncake/GPUCompiler (FunctionWrappers 1.1.2, PrecompileTools 1.0.0, TruncatedStacktraces 1.1.0, SafeTestsets 0.1.0, SciMLTesting 1.0.0). - Pkg.test GROUP=Core with allow_reresolve=false, force_latest_compatible_version=false (the julia-runtest CI settings): precompiles cleanly and passes (FunctionWrappersWrappers.jl 48/48, BigFloat + UnionAll 7/7). - The AD groups still run via their sub-envs: GROUP=Enzyme 65/65, GROUP=Mooncake 13/13 (Julia 1.11), with the Enzyme/Mooncake extensions precompiling in-group. Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The
Downgrade (Core)job was red. The root cause is that the AD test extras (Enzyme,EnzymeCore,Mooncake) were part of the root test target ([targets].test), so the downgrade resolver had to place them in the minimum-version test env.julia-downgrade-compat(Resolver.jl) minimizes Julia to the floor of its--juliarange; onltsthat admits1.10.0. But everyMooncake 0.5.xrequiresjulia >= 1.10.8, and theEnzyme 0.13+EnzymeCore 0.8+Mooncake 0.5minimum-version set has no solution on the 1.10 LTS stdlib line, so the merged test env isUnsatisfiable. This is the Resolver.jl <-> Mooncake wall (StefanKarpinski/Resolver.jl#24). The AD deps should not gate the downgrade lane.Fix: split the AD group out of downgrade (instead of pinning Julia)
test/Enzyme/Project.tomlandtest/Mooncake/Project.toml, each with[sources] FunctionWrappersWrappers = {path = "../.."}. DropEnzyme/EnzymeCore/Mooncakefrom the root[extras]/[targets].test. (They stay as[weakdeps]+[compat]for the extensions.)test/runtests.jl: theEnzyme/Mooncakegroups now use SciMLTesting's per-groupenv = ...form (the same mechanism thenopregroup already uses), so each runs in its own env.test/test_groups.toml: addEnzymeandMooncakegroups (["lts", "1"]). They were declared inruntests.jlbut had no matrix entry, so they were not running in CI at all — now they do.Downgradejob runsGROUP=Core, which resolves only the root test target — now AD-free — so it is satisfiable onlts(Julia 1.10) again.Reverted #59 accommodations (they masked the AD-resolution problem)
.github/workflows/Downgrade.yml:julia-versionback to"lts"(no longer pinned to1.11).PrecompileToolscompat floor back to"1": the1.0.0GPUCompiler precompile failure only arose because Enzyme pulled GPUCompiler into the downgrade env; with AD out of that env it no longer applies.Kept — genuine minimum-version bugs, independent of AD (verified on Julia 1.10)
TruncatedStacktraces = "1.1":srcuses@truncate_stacktrace, which does not exist before v1.1.0 (1.0.0 is a 25-line stub).FunctionWrappers = "1.1.2": 1.0.0 / 1.1.0 / 1.1.1 fail to build theFunctionWrappercfunction (Module IR does not contain specified entry function) on Julia 1.10; 1.1.2 is the lowest that builds and runs. This is exercised by the package's own PrecompileTools@compile_workload, so it is a real floor regardless of the AD split. (This vindicates the earlierFunctionWrappersfloor raise — the floor is real; only the exact version was off, 1.1.2 vs 1.1.3.)Local verification (Julia 1.10 / lts — the actual downgrade leg)
Ran the real
julia-actions/julia-downgrade-compat@v2downgrade.jl(modedeps, projects., effective skip = Julia stdlibs):Successfully resolved minimal versions for . (with extras).Enzyme/Mooncake/GPUCompiler. Floors:FunctionWrappers 1.1.2,PrecompileTools 1.0.0,TruncatedStacktraces 1.1.0,SafeTestsets 0.1.0,SciMLTesting 1.0.0.Pkg.testGROUP=Corewithallow_reresolve=false, force_latest_compatible_version=false(thejulia-runtestCI settings): precompiles cleanly and passes —FunctionWrappersWrappers.jl48/48,BigFloat + UnionAll7/7.AD groups still run via their sub-envs (Julia 1.11):
GROUP=Enzyme65/65,GROUP=Mooncake13/13, withFunctionWrappersWrappersEnzymeExt/FunctionWrappersWrappersMooncakeExtprecompiling in-group. The grouped-tests matrix script now emits 9 jobs (Core lts/1/pre, Enzyme lts/1, Mooncake lts/1, nopre lts/1).Please ignore until reviewed by @ChrisRackauckas